Don't embed external initializers into the proto to avoid 2GB limit #817

intbf · 2025-09-19T20:37:23Z

Description

When calling subgraph.ToProto don't save initializers, so that we don't bloat the model proto. This is especially important for models with ext initializers in memory that in total contain more than 2GB of data.

Once we have the list of those external initializers, we can update ther metadata inside the proto, so that they leverage the special marker: */_ORT_MEM_ADDR_/*. (The support for this location was recently fixed inside OV with openvinotoolkit/openvino#31632)

Motivation and Context

CVS-173057, CVS-172710

Todo/questions

Should we have some condition before using this approach? Maybe when model has those ext weights?
The code in GetModelProtoFromFusedNode currenly supports only one case, the regular case, but similar logic have to be applied to other cases like qdq stripping, scale fix, etc.
Add unit tests

Copilot

Pull Request Overview

This PR modifies the model proto generation to avoid embedding external initializers directly into the proto, preventing potential issues when models exceed the 2GB protobuf limit. The changes introduce a mechanism to handle initializers with external data by updating their metadata to use a special memory address marker instead of including the actual data.

Key changes:

Add logic to exclude initializer data from proto generation when load_user_initializer_ is enabled
Implement metadata updating for external initializers using the */_ORT_MEM_ADDR_/* marker
Add comprehensive logging for debugging initializer processing

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

onnxruntime/core/providers/openvino/backend_manager.cc

Copilot

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

onnxruntime/core/providers/openvino/backend_manager.cc

Copilot

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 5 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

onnxruntime/core/providers/openvino/backend_manager.cc

Copilot · 2025-09-30T12:23:35Z

onnxruntime/core/providers/openvino/backend_manager.cc

-    subgraph.ToProto(*model_proto->mutable_graph(), true, true);
+    subgraph.ToProto(*model_proto->mutable_graph(), /*include_initializers*/true, 
+                     /*include_outer_scope_args*/true, /*execution_order*/0, /*include_initializer_data*/include_initializer_data_in_proto);
+


Remove the trailing whitespace on this empty line.

Suggested change

onnxruntime/core/providers/openvino/backend_manager.cc

… restores the metadata so OV can read them back Signed-off-by: bfilipek <[email protected]>

…re are external initializers in memory (more than one) Signed-off-by: bfilipek <[email protected]>

Signed-off-by: bfilipek <[email protected]>

…t initializers, comments, refactoring Signed-off-by: bfilipek <[email protected]>

onnxruntime/core/providers/openvino/backend_manager.cc

Co-authored-by: Copilot <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

MayureshV1 · 2025-10-02T00:43:06Z

openvinotoolkit/openvino#31632 was only added after OV in OV25.3, so we need to have a OV version check implemented.

…r the logic is executed, check for OV version Signed-off-by: bfilipek <[email protected]>

Signed-off-by: bfilipek <[email protected]>

…ver 2GB to show the proto limit (when the new logic for ext initializers is enabled, then the test passes) Signed-off-by: bfilipek <[email protected]>

onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc

Signed-off-by: bfilipek <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc

Copilot · 2025-10-04T00:16:49Z

onnxruntime/core/providers/openvino/backend_manager.cc

+    const bool include_initializer_data_in_proto = !(session_context_.has_external_weights && 
+                                                     external_initializers_offset_and_length.size() > 1 && 


The condition external_initializers_offset_and_length.size() > 1 seems arbitrary. Consider using a named constant or documenting why specifically more than 1 external initializer is required to trigger this optimization.

Suggested change

const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&

external_initializers_offset_and_length.size() > 1 &&

// Optimization is only triggered if there is more than one external initializer,

// as the benefit of excluding data from the proto is only significant in that case.

constexpr size_t MIN_EXTERNAL_INITIALIZERS_FOR_OPTIMIZATION = 2;

const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&

external_initializers_offset_and_length.size() >= MIN_EXTERNAL_INITIALIZERS_FOR_OPTIMIZATION &&

Copilot · 2025-10-04T00:16:49Z

onnxruntime/core/providers/openvino/backend_manager.cc

+
+          LOGS(logger, VERBOSE) << "In-memory initializer EXT: " << src_init->name() << ", size: " << length;
+
+          SetExternalDataFields(proto_init, (const void*)offset, length);


Casting offset (which is a size_t representing a memory offset) directly to (const void*) is incorrect. The offset should be added to a base pointer to get the actual memory address, not used as a pointer itself.

Suggested change

SetExternalDataFields(proto_init, (const void*)offset, length);

SetExternalDataFields(proto_init, static_cast<const void*>(static_cast<const uint8_t*>(external_initializers_data) + offset), length);

Co-authored-by: Copilot <[email protected]>

preetha-intel

LGTM

Signed-off-by: bfilipek <[email protected]>

…nt CI version is only 2025.2 Signed-off-by: bfilipek <[email protected]>

Signed-off-by: bfilipek <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-06T17:05:21Z

onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc

+namespace onnxruntime {
+namespace test {
+
+// this test requiresOV 2025.4+ to run, currently CI uses OV 2025.2, so the test will be disabled until OV is updated


Missing space in 'requiresOV' - should be 'requires OV'.

Suggested change

// this test requiresOV 2025.4+ to run, currently CI uses OV 2025.2, so the test will be disabled until OV is updated

// this test requires OV 2025.4+ to run, currently CI uses OV 2025.2, so the test will be disabled until OV is updated

Copilot · 2025-10-06T17:05:22Z

onnxruntime/core/providers/openvino/backend_manager.cc

+    subgraph.ToProto(*model_proto->mutable_graph(), /*include_initializers*/true, 
+                     /*include_outer_scope_args*/true, /*execution_order*/0, /*include_initializer_data*/include_initializer_data_in_proto);


The execution_order parameter is hardcoded to 0. Consider using a named constant or adding a comment explaining why 0 is the appropriate default value.

MayureshV1

LGTM !

… limit (#817)" This reverts commit f8b0904.

javier-intel requested review from MayureshV1, ericcraw, javier-intel and Copilot September 23, 2025 13:59

Copilot AI reviewed Sep 23, 2025

View reviewed changes

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

preetha-intel self-requested a review September 24, 2025 04:41

intbf force-pushed the dont_write_ext_initializers branch from fd29454 to 2b34773 Compare September 26, 2025 13:22

intbf requested a review from Copilot September 26, 2025 19:23

Copilot AI reviewed Sep 26, 2025

View reviewed changes

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

intbf commented Sep 27, 2025

View reviewed changes

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

intbf requested a review from Copilot September 30, 2025 12:22

Copilot AI reviewed Sep 30, 2025

View reviewed changes

intbf marked this pull request as ready for review September 30, 2025 12:23

intbf added 4 commits October 1, 2025 00:23

early version, it doesn't embed initializers into the proto, but then…

30e9d5f

… restores the metadata so OV can read them back Signed-off-by: bfilipek <[email protected]>

improve code, refactor into smaller functions, run the logic when the…

70c2f37

…re are external initializers in memory (more than one) Signed-off-by: bfilipek <[email protected]>

revert the wrongly merged code

80de8ef

Signed-off-by: bfilipek <[email protected]>

Updated the condition for the new logic based on the total size of ex…

ef6f23d

…t initializers, comments, refactoring Signed-off-by: bfilipek <[email protected]>

intbf force-pushed the dont_write_ext_initializers branch from 7907cd7 to ef6f23d Compare October 1, 2025 07:23

ericcraw reviewed Oct 1, 2025

View reviewed changes

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

javier-intel approved these changes Oct 1, 2025

View reviewed changes

MayureshV1 requested a review from Copilot October 2, 2025 00:32

Update onnxruntime/core/providers/openvino/backend_manager.cc

165a661

Co-authored-by: Copilot <[email protected]>

Copilot AI reviewed Oct 2, 2025

View reviewed changes

intbf added 3 commits October 2, 2025 14:00

make the condition less strict - 32MB threshold, move debug dump afte…

fe2cf8c

…r the logic is executed, check for OV version Signed-off-by: bfilipek <[email protected]>

unit test that uses ext initializers, early version

e6727b1

Signed-off-by: bfilipek <[email protected]>

used kOrtSessionOptionsDisableCPUEPFallback, cleanups, model is now o…

033b6f9

…ver 2GB to show the proto limit (when the new logic for ext initializers is enabled, then the test passes) Signed-off-by: bfilipek <[email protected]>

ericcraw approved these changes Oct 3, 2025

View reviewed changes

onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc Outdated Show resolved Hide resolved

onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc Outdated Show resolved Hide resolved

address code review comments

d4e41c9

Signed-off-by: bfilipek <[email protected]>

MayureshV1 requested a review from Copilot October 4, 2025 00:16

Copilot AI reviewed Oct 4, 2025

View reviewed changes

Update onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc

cba53b3

Co-authored-by: Copilot <[email protected]>

preetha-intel approved these changes Oct 6, 2025

View reviewed changes

intbf added 3 commits October 6, 2025 00:38

fix the Linux CI build, use PathString rather than wstring

8a5fe0e

Signed-off-by: bfilipek <[email protected]>

As agreed, disable the test as it requires OV 2025.4, while the curre…

2a0fa0a

…nt CI version is only 2025.2 Signed-off-by: bfilipek <[email protected]>

add missing comment

a345737

Signed-off-by: bfilipek <[email protected]>

MayureshV1 requested a review from Copilot October 6, 2025 17:04

Copilot AI reviewed Oct 6, 2025

View reviewed changes

MayureshV1 approved these changes Oct 6, 2025

View reviewed changes

MayureshV1 merged commit f8b0904 into intel:ovep-develop Oct 6, 2025
3 of 5 checks passed

MayureshV1 added a commit that referenced this pull request Oct 7, 2025

Revert "Don't embed external initializers into the proto to avoid 2GB…

7eafcb3

… limit (#817)" This reverts commit f8b0904.

intbf mentioned this pull request Oct 8, 2025

Fix Regression in Model PSS and PSR, add check for zero-size initializers #825

Merged

		const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&
		external_initializers_offset_and_length.size() > 1 &&

-    const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&
-                                                     external_initializers_offset_and_length.size() > 1 &&
+    // Optimization is only triggered if there is more than one external initializer,
+    // as the benefit of excluding data from the proto is only significant in that case.
+    constexpr size_t MIN_EXTERNAL_INITIALIZERS_FOR_OPTIMIZATION = 2;
+    const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&
+                                                     external_initializers_offset_and_length.size() >= MIN_EXTERNAL_INITIALIZERS_FOR_OPTIMIZATION &&


		LOGS(logger, VERBOSE) << "In-memory initializer EXT: " << src_init->name() << ", size: " << length;

		SetExternalDataFields(proto_init, (const void*)offset, length);

	SetExternalDataFields(proto_init, (const void*)offset, length);
	SetExternalDataFields(proto_init, static_cast<const void>(static_cast<const uint8_t>(external_initializers_data) + offset), length);

	// this test requiresOV 2025.4+ to run, currently CI uses OV 2025.2, so the test will be disabled until OV is updated
	// this test requires OV 2025.4+ to run, currently CI uses OV 2025.2, so the test will be disabled until OV is updated

		subgraph.ToProto(model_proto->mutable_graph(), /include_initializers*/true,
		/include_outer_scope_args/true, /execution_order/0, /include_initializer_data/include_initializer_data_in_proto);

Don't embed external initializers into the proto to avoid 2GB limit #817

Don't embed external initializers into the proto to avoid 2GB limit #817

Uh oh!

Conversation

intbf commented Sep 19, 2025

Description

Motivation and Context

Todo/questions

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

MayureshV1 commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI Oct 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 4, 2025

Choose a reason for hiding this comment

Uh oh!

preetha-intel left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

MayureshV1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MayureshV1 commented Oct 2, 2025 •

edited

Loading